Measuring Semantic Coverage

The developlnent of natural language processing systems is currently driven to a large extent by measures of knowledgebase size and coverage of individual phenomena relative to a corpus. While these measures have led to significant advances for knowledge-lean applications, they do not adequately motivate progress in computat ional semantics leading to the development of large-scale, general purpose NLP systems. In this article, we argue tha t depth of semantic representat ion is essential for covering a broad range of phenomena in the computational t rea tment of language and propose (lepth as an impor tan t additional dimension for measuring the semantic coverage of NLP systems. We propose an operationalization of this measure and show how to characterize an NLP system along the dimensions of size, corpus coverage, and depth. The proposed framework is illustrated using sever~fl prominent NLP systems. We hope the preliminary proposals made in this article will lead to prolonged debates in the field and will continue to be refined. 1 Measures of Size versus Measures of Depth Evaluation of current and potential performance of' an NLP system or method is of crucial importance to researchers, developers and users. Current performance of systems is directly measured using a variety of tests and techniques. Often, as in the case of machine translation or information extraction, an entire "industry" of evaluation gets developed (see, for example, ARPA MT Evaluation; MUC-4 ). Measuring the performance of an NLP method, approach or technique (and through it the promise of a system based on it) is more difficult, as judgments must be made about "blame assigmnent" and the impact of improving a variety of system components on the overall future performance. One of the widely accepted measures of potential performance improvement is the feasibility of scaling up the static knowledge sources of an NLP system its grammars , lexicons, worht knowledge bases and other sets of language descriptions (the reasoning being that the larger the system's grammars and lexicons, the greater percentage of input they would be able to match and, therefore, the better the performance of the systeml) . As a result, a system would be considered very promising if its knowledge sources could be significantly scaled up at a reasonable expense. Natm'ally, the expense is lowest if acquisition is performed automatically. This consideration and the recent resurgence of corpus-based methods heighten the interest in the automat ion of knowledge acquisition, llowever, we believe that such acquisition should not 1)e judged solely by the utility of acquired knowledge for ~ particular application. A preliminary to the sealability estimates is a judgment of the current coverage of a system's static knowledge sources. Unfortunately, judgments based purely on size ace often misleading. While they may be sufficiently straightforward for less km)wledgeAntensive methods used in such applications as information extraction and retrieval, part of speech tagging, bilingual corpus alignment, and so on, the saute is not true about more ruleand knowledge-based methods (such as syntactic parsers, semantic analyzers, semantic lexicons, ontological world models, etc.). It ix widely accepted, for instance, that judgments of the coverage of a syntactic g rammar in terms of the number of rules are tlawed. It is somewhat less self-evident, however, that the number of lexicon entries or ontology concepts is not an adequate measure of the quality or coverage of NLP a Incidentally, this consideration eontributes to evMuation of current perforntance as well. In the absence of actual evaluation results, it is customary to c|aim the utility of the system by simply mentioning tit(: size of its knowledge sources (e.g., "over 550 grammar rules, over 50,000 concepts in the ontology and over 100,00(I word senses in the dictionary").